This invention relates to an image processing method and apparatus and an image display method and apparatus, more particularly to methods and apparatus that are useful for expanding and reducing digitized images, and for controlling edge sharpness.
Digital image expansion and reduction are processes that change the number of picture elements (pixels) in an image. These processes are often necessary. For example, an image generated in the widely used 640-by-480-pixel format may need to be expanded for display on a 1024-by-768-pixel liquid crystal screen, or reduced for display in a window occupying only part of that screen.
FIGS. 1A and 1B illustrate the conventional expansion of an image by a factor of three. The horizontal axis in these drawings represents a horizontal row of pixels in the image; the vertical axis represents the pixel data values, indicating pixel brightness levels. Before expansion, the row of pixels has segments (h) of uniform brightness, separated by edges (j, k) at which the brightness level changes, as shown in FIG. 1A. The conventional expansion process expands all of these segments and edges identically by a factor of three, so that the edges (j1, k1) in the expanded image, shown in FIG. 1B, are not as sharp as the edges in the original image.
The pixel values in the expanded picture are determined by interpolation, which is performed by a spatial filtering operation as illustrated in FIG. 2. The horizontal axis again represents horizontal position; the vertical axis now represents the value of an interpolation filter response characteristic F(x). If p(n) and p(n+1) are two consecutive pixels in the input image, q(n) is a pixel disposed at a position between them in the output image, the distance from p(n) to p(n+1) is equal to unity, and the distance from p(n) to q(m) is equal to r, then the brightness level of pixel q(m) is calculated as follows:
q(m)=(F(r)xc3x97p(n))+(F(1xe2x88x92r)xc3x97p(n+1))
FIG. 3 illustrates schematically how this filtering calculation generates seven output pixels (q1 to q7) from three input pixels (p1 to p3).
The filter response characteristic need not be linear. An interpolation filter with a nonlinear characteristic, as illustrated in FIG. 4, is sometimes used to enhance the sharpness of edges (e.g., j1 and k1) in the output image. This type of edge enhancement, however, leads to further problems such as undershoot (pre-shoot) and overshoot.
Japanese Unexamined Patent Application No. 9-266531 discloses a scheme that provides several filter response characteristics, as illustrated in FIGS. 5A, 5B, and 5C, and switches among them according to the type of image area being processed. When this filter-switching scheme is used, however, there are problems of discontinuities at the points at which switching takes place.
Problems also occur when an image is reduced by conventional methods. The quality of edges is degraded because of pixel dropout.
One object of this invention is to mitigate edge degradation when an image is expanded or reduced.
Another object is to control edge sharpness in an image.
The invention provides a method of processing an image, formed from input pixels, by the following steps:
(a) detecting pixel-to-pixel variations in the brightness levels of the input pixels in at least one direction in the image, thereby generating high spatial frequency information;
(b) setting interpolation points with a spacing that varies according to the high spatial frequency information; and
(c) generating output pixels from the input pixels by interpolation at the interpolation points.
Step (b) preferably assigns a basic value to the spacing of the interpolation points in parts of the image in which the brightness level is uniform. As for portions of the image in which the brightness level varies, step (b) may divide each such portion into a first part and a second part, reduce the interpolation spacing in the first part, and increase the interpolation spacing in the second part. Alternatively, step (b) may divide each such portion into a first part, a second part, and a third part, reduce the interpolation spacing in the first and third parts, and increase the interpolation spacing in the second part.
Step (a) includes, for example, calculating a first derivative of the brightness levels in the above-mentioned one direction. Step (a) may also include calculating a second or third derivative of the brightness levels, or performing a filtering operation to obtain a certain spatial frequency component of the image, by performing two low-pass filtering operations with different cut-off frequencies and taking the difference between the resulting two low spatial frequency components, for example.
Alternatively, step (a) may include detecting patterns of variation in the brightness levels of the input pixels. The detected patterns describe, for example, the polarity of pixel-to-pixel changes in brightness level, or the polarity and magnitude of the changes. The changes thus described may be, for example, changes in the brightness levels of three consecutive pixels, or changes in the brightness levels of five consecutive pixels.
The invention also provides a machine-readable medium storing a machine-executable program for processing an image by the invented method.
The invention moreover provides an image-processing apparatus for processing an image formed from input pixels. The apparatus includes a first processing unit that detects pixel-to-pixel variations in the brightness levels of the input pixels in at least one direction in the image, and generates high spatial frequency information; a second processing unit that sets interpolation points with a spacing varying according to the high spatial frequency information; and a third processing unit that generates output pixels from the input pixels by interpolation at the interpolation points.
The second processing unit preferably assigns a basic value to the interpolation spacing in parts of the image in which the brightness level is uniform. In processing portions of the image in which the brightness level changes, the second processing unit may divide each such portion into a first part and a second part, reduce the interpolation spacing in the first part, and increase the interpolation spacing in the second part. Alternatively, it may divide each such portion into a first part, a second part, and a third part, reduce the interpolation spacing in the first and third parts, and increase the interpolation spacing in the second part.
As high spatial frequency information, the first processing unit may calculate a first derivative of the brightness levels in the above-mentioned one direction. The first processing unit may also calculate a second or third derivative of the brightness levels, or perform a filtering operation to obtain a certain spatial frequency component of the image, by performing two low-pass filtering operations with different cut-off frequencies and taking the difference between the resulting two low spatial frequency components, for example.
Alternatively, the first processing unit may detect patterns of variation in the brightness levels of the input pixels. The detected patterns describe, for example, the polarity of pixel-to-pixel changes in brightness level, or the polarity and magnitude of the changes. The changes thus described may be, for example, changes in the brightness levels of three consecutive pixels, or changes in the brightness levels of five consecutive pixels.
The invention furthermore provides an image display apparatus for displaying an image formed from input pixels. The image display apparatus includes a memory unit that stores the brightness levels of the input pixels; a first processing unit that detects pixel-to-pixel variations in the brightness levels in at least one direction in the image, thereby generating high spatial frequency information; a second processing unit that sets interpolation points with a spacing that varies according to the high spatial frequency information; a third processing unit that generates output pixels from the input pixels by interpolation at the interpolation points; and a display unit that displays the output pixels.
The second processing unit preferably assigns a basic value to the interpolation spacing in parts of the image in which the brightness level is uniform. In processing portions of the image in which the brightness level changes, the second processing unit may divide each such portion into a first part and a second part, reduce the interpolation spacing in the first part, and increase the interpolation spacing in the second part. Alternatively, it may divide each such portion into a first part, a second part, and a third part, reduce the interpolation spacing in the first and third parts, and increase the interpolation spacing in the second part.
As high spatial frequency information, the first processing unit may calculate a first derivative of the brightness levels in the above-mentioned one direction. The first processing unit may also calculate a second or third derivative of the brightness levels, or perform a filtering operation to obtain a certain spatial frequency component of the image, by performing two low-pass filtering operations with different cut-off frequencies and taking the difference between the resulting two low spatial frequency components, for example.
Alternatively, the first processing unit may detect patterns of variation in the brightness levels of the input pixels. The detected patterns describe, for example, the polarity of pixel-to-pixel changes in brightness level, or the polarity and magnitude of the changes. The changes thus described may be, for example, changes in the brightness levels of three consecutive pixels, or changes in the brightness levels of five consecutive pixels.
Varying the spacing of the interpolation points provides a way to control edge sharpness without introducing discontinuities or other unwanted image artifacts.
Increasing the interpolation spacing mitigates the loss of edge sharpness that occurs when an image is expanded.
Decreasing the interpolation spacing mitigates the drop-out problem that occurs when an image is reduced.
Dividing an image portion into multiple parts and increasing the interpolation spacing in at least one part while decreasing the interpolation spacing in another part enables the loss-of-sharpness problem to be mitigated in edge expansion, and the drop-out problem to be mitigated in image reduction, without changing the overall expansion or reduction ratio.
Use of the first, second, and third derivatives as high spatial frequency information enables continuously-varying information to be obtained by simple arithmetic operations. Filtering also enables such information to be obtained by relatively simple arithmetic operations.
Detecting patterns of variation in the brightness levels of the input pixels enables high-frequency information to be generated by simple arithmetic and logic operations. Detecting patterns in the polarity of changes over three consecutive pixels provides adequate continuity, but the continuity can be improved by also detecting the magnitude of the changes, or by detecting patterns over five consecutive pixels.